Make active-learning runs reproducible under a fixed seed#200
Open
CoolJosh0221 wants to merge 3 commits into
Open
Make active-learning runs reproducible under a fixed seed#200CoolJosh0221 wants to merge 3 commits into
CoolJosh0221 wants to merge 3 commits into
Conversation
Audit of every randomness entry point (strategy tie-breaking, QBC/BALD bootstrap bagging, diversity/clustering selectors, model init paths, labeler, dataset utilities, and the C extensions - which are RNG-free) found three places not controlled by a passed seed: - IdealLabeler.label drew from the global np.random with no way to seed it; it now accepts random_state and breaks ties between duplicate features with conflicting labels reproducibly. - Dataset.labeled_uniform_sample sampled via the global np.random; it now accepts an optional random_state. - import_scipy_mat shuffled entries (and hence all positional entry ids) via the global np.random; it now accepts an optional random_state. A new libact.utils.check_random_state helper centralizes the seeding rule for these call sites: None keeps the global numpy random state so unseeded behavior is byte-identical to before, while an int or RandomState fully determines the outcome. seed_random_state now also accepts numpy integer scalars as seeds. Adds test_seed_reproducibility.py: two full active-learning runs with the same seed must produce identical query sequences across 15 strategies (including QBC bootstrapping, BALD ensembles, epsilon-greedy exploration, clustering-based and C-extension-backed strategies, and ALBL), plus a canonical loop with an IdealLabeler over conflicting-duplicate data, and seeding tests for the dataset utilities. Mutation-checked: re-introducing global-RNG draws in IdealLabeler or the QBC bootstrap sampler makes the new tests fail.
BALD's base_model + n_models path manufactures its committee via clone(), so a stochastic base estimator (e.g. RandomForestClassifier) left the run non-reproducible even when a seed was passed to BALD, and seeding the base estimator directly collapsed committee diversity to identical members. BALD now derives a distinct deterministic child seed per clone from its random_state_ (mirroring scikit-learn's ensemble behavior), and SklearnAdapter/SklearnProbaAdapter.clone() gained an optional random_state that reseeds the cloned estimator when it exposes one. No-arg clone() is unchanged; estimators without a random_state parameter and custom clone() signatures are handled gracefully. Adds test_bald_base_model_stochastic covering the stochastic base_model path; it fails against the previous unseeded clone construction.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A passed
random_statenow fully determines a run's query sequence. Audited every randomness source (strategy tie-breaking, QBC/BALD bootstrapping, KMeans-based strategies, model init, labeler, dataset utilities, and the HintSVM/variance-reduction C extensions — which contain no RNG calls). Four sites were not governed by the passed seed:IdealLabeler.labelcalled module-levelnp.random.choice(the global NumPy random state) and had norandom_stateparameter — now accepts one.Dataset.labeled_uniform_sample(np.random.choice) andimport_scipy_mat(np.random.shuffle) drew from the global NumPy random state — both now accept an optionalrandom_state.base_modelcommittee is built withsklearn.clone, which copies the base estimator'srandom_stateverbatim. So BALD's seed controlled the bootstrap bags but not each clone's internal fit randomness: a stochastic base (e.g.RandomForestClassifier(random_state=None)) stayed non-reproducible, while seeding the base with a fixed int made all clones identical. Each clone now receives a distinct deterministic seed derived from BALD'srandom_state_.New
libact.utils.check_random_statecentralizes the rule for the labeler/dataset sites:Nonereturns the global NumPy random state, so unseeded behavior is byte-identical to before; an int orRandomStatereturns a seeded instance.seed_random_statealso accepts numpy integer scalars.Tests
test_seed_reproducibility.py: same seed ⇒ identical query sequence across 14 query strategies (16 same-seed cases, incl. QBC bootstrapping, BALD via bothmodels=and stochasticbase_model=, ε-greedy, KMeans-based, ALBL), plus anIdealLabelerloop over conflicting-duplicate rows and dataset-utility seeding. Mutation-checked: reverting any of the four fixes fails its test. Unseeded behavior unchanged; full suite green.Scope
The three global-RNG sites were already reproducible via
np.random.seed()(as the existing tests do); this mostly makes a passedrandom_statesufficient and composable, and adds a test guard so the property can't silently regress. BALD is the exception — a committee that is both reproducible and diverse from a passed seed wasn't achievable before.